<html>
<head><meta charset="utf-8"><title>logging infrastructure for services · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html">logging infrastructure for services</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="236971745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236971745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236971745">(May 01 2021 at 14:25)</a>:</h4>
<p>So, we have Prometheus for metrics; do we have a solution for logging?</p>
<blockquote>
<p>Complementing metrics are event logs, which tend to be more delayed and have fewer pieces of information, but no cardinality limitations.</p>
</blockquote>
<p>Some back-of-the-envelope calculations show that I <em>might</em> be pushing the limits of cardinality for the playground metrics I want to add. I think I'll still go forward with it for now to get a handle on the current problem, but might dial it back down later when we have figured things out.</p>
<p>Do we have a nice way of aggregating application logs and gleaming information out of them?</p>



<a name="236971846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236971846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236971846">(May 01 2021 at 14:26)</a>:</h4>
<p>Not that I know of, but while I don't know the details I'd be surprised if you're actually pushing cardinality limits or anything like that</p>



<a name="236971978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236971978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236971978">(May 01 2021 at 14:28)</a>:</h4>
<p>and if you are, then the better solution is likely to split up the metrics into different names, I suspect.</p>



<a name="236972002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972002">(May 01 2021 at 14:28)</a>:</h4>
<p>I believe the cardinality will be 3,810,240</p>



<a name="236972014"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972014" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972014">(May 01 2021 at 14:29)</a>:</h4>
<p>that seems.. indeed high</p>



<a name="236972015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972015">(May 01 2021 at 14:29)</a>:</h4>
<p>what are you storing?</p>



<a name="236972017"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972017" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972017">(May 01 2021 at 14:29)</a>:</h4>
<p>That's maximum, I'm sure the actual will be less</p>



<a name="236972028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972028">(May 01 2021 at 14:29)</a>:</h4>
<p>e.g. certain combinations cannot possibly happen</p>



<a name="236972125"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972125" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972125">(May 01 2021 at 14:30)</a>:</h4>
<p>hm well in practice I'd guess your cardinality is like, a couple dozen?</p>



<a name="236972153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972153">(May 01 2021 at 14:30)</a>:</h4>
<p>it's worth noting that I would personally start with <em>less</em> rather than more, but ultimately it may not matter that much</p>



<a name="236972173"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972173" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972173">(May 01 2021 at 14:31)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services/near/236971978">said</a>:</p>
<blockquote>
<p>and if you are, then the better solution is likely to split up the metrics into different names, I suspect.</p>
</blockquote>
<p>I don't know how that would help. As I understand it, ultimately each unique label set makes a different metric, under the hood.</p>



<a name="236972195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972195">(May 01 2021 at 14:31)</a>:</h4>
<p>sure, I mean, like, ultimately you need to track what you need to track. but your max cardinality would surely go down if you're not multiplying llvm by rustfmt etc</p>



<a name="236972338"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972338" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972338">(May 01 2021 at 14:33)</a>:</h4>
<p>I guess that's a matter of max theoretical vs actual then. e.g. is <code>thing{llvm="yes", asm=""} thing{llvm="", asm="yes"}</code> any different from <code>thing_llvm{} </code>thing_asm{}` in practice.</p>



<a name="236972390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972390">(May 01 2021 at 14:34)</a>:</h4>
<p>I don't know in terms of underlying storage but I'd sort of guess no, or at least, not enough to matter</p>



<a name="236972438"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972438" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972438">(May 01 2021 at 14:34)</a>:</h4>
<p>theoretically the first would allow for cardinality of 4, but two of those wouldn't happen</p>



<a name="236972441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972441">(May 01 2021 at 14:34)</a>:</h4>
<p>but the second is potentially easier to deal with in grafana</p>



<a name="236972464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972464">(May 01 2021 at 14:34)</a>:</h4>
<p>in any case, that seems off - i would expect build_kind = {llvm,asm,rustc,miri,...}</p>



<a name="236972466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972466">(May 01 2021 at 14:35)</a>:</h4>
<p>not sure if just bad example</p>



<a name="236972471"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972471" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972471">(May 01 2021 at 14:35)</a>:</h4>
<p>sigh. I thought Pietro said the opposite, which is why I rewrote the metrics a third time.</p>



<a name="236972480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972480">(May 01 2021 at 14:35)</a>:</h4>
<p>just a quick example</p>



<a name="236972508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972508">(May 01 2021 at 14:35)</a>:</h4>
<p>hmm</p>



<a name="236972529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972529">(May 01 2021 at 14:35)</a>:</h4>
<p>labels as booleans seems bad, basically, is my intuition</p>



<a name="236972531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972531">(May 01 2021 at 14:35)</a>:</h4>
<p>Here's the dump of my local dev after clicking a bit <a href="https://gist.github.com/shepmaster/139e95d35ccb74caf6da30adc1d510d5">https://gist.github.com/shepmaster/139e95d35ccb74caf6da30adc1d510d5</a></p>



<a name="236972621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972621">(May 01 2021 at 14:36)</a>:</h4>
<p>I would probably move the meta endpoints (which never have most of the labels set) to their own 'metric'</p>



<a name="236972640"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972640" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972640">(May 01 2021 at 14:37)</a>:</h4>
<p>at the very least, it seems odd to have no channel but the channel embedded in endpoint name</p>



<a name="236972649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972649">(May 01 2021 at 14:37)</a>:</h4>
<p>ah, though I see, MetaVersionRustfmt</p>



<a name="236972667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972667">(May 01 2021 at 14:37)</a>:</h4>
<p>yeah anyway, this dump seems ok to start with, we can always iterate</p>



<a name="236972676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972676">(May 01 2021 at 14:38)</a>:</h4>
<p>ultimately it's no big deal to try things I suspect</p>



<a name="236972741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972741">(May 01 2021 at 14:38)</a>:</h4>
<p>There's a (closed?) PR that consolidates all the meta into one endpoint that I should merge someday anyway.</p>



<a name="236972765"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972765" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972765">(May 01 2021 at 14:39)</a>:</h4>
<p>Because you almost always want all of them, and that reduces the roundtrips</p>



<a name="236972770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972770">(May 01 2021 at 14:39)</a>:</h4>
<p>Yeah.</p>



<a name="236972900"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236972900" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236972900">(May 01 2021 at 14:42)</a>:</h4>
<p>Say that I did something totally stupid and flooded prometheus with billions of bad data — is there a "delete these things but keep everything else" solution to that?</p>



<a name="236973132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973132">(May 01 2021 at 14:45)</a>:</h4>
<p>hm</p>



<a name="236973137"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973137" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973137">(May 01 2021 at 14:45)</a>:</h4>
<p>not that I know of, in some sense</p>



<a name="236973151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973151">(May 01 2021 at 14:45)</a>:</h4>
<p>I guess we have some recent-ish backups, or so</p>



<a name="236973218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973218">(May 01 2021 at 14:46)</a>:</h4>
<p>but we can also have the metrics endpoint exposed and not immediately attach prometheus, so you can just look at the http results (obviously can test locally too)</p>



<a name="236973230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973230">(May 01 2021 at 14:46)</a>:</h4>
<p>the number of metrics should, I expect, be pretty much static - if not, then usually something is off</p>



<a name="236973779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236973779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236973779">(May 01 2021 at 14:55)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116155">Jake Goulding</span> <a href="#narrow/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services/near/236972900">said</a>:</p>
<blockquote>
<p>Say that I did something totally stupid and flooded prometheus with billions of bad data — is there a "delete these things but keep everything else" solution to that?</p>
</blockquote>
<p>prometheus has an api to delete metrics</p>



<a name="236976183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/logging%20infrastructure%20for%20services/near/236976183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/logging.20infrastructure.20for.20services.html#236976183">(May 01 2021 at 15:11)</a>:</h4>
<p>Ok, if either of you want to review, it's up a <a href="https://github.com/integer32llc/rust-playground/pull/714">https://github.com/integer32llc/rust-playground/pull/714</a>. I can't run CI now because rustfmt is missing from nightly, so no immediate rush.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>